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ABSTRACT 


The appropriateness of a multidimensional scaling tech- 
Migue (MDS) in attempting to quantify students perceptions 
of teacher performance is investigated inthe following 
study. Data collected on an interactive computer survey and 
from Student Opinion Forms (SOF's) are used to determine if 
Satisfactory linear relationships for teacher performance 
exist. Multiple linear regression and factor analysis 
attempt to identify what appear to be the most important 
characteristics in instructor performance according to the 


perceptions of a control group. Spatial plots are created 
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I. INTRODUCTION 


ie SUREPOS S 


The purpose of this thesis is twofold: 

1. To develop methodology that will help discover the 
important characteristics of instructor performance 
aS perceived by each student group; 

2. To develop user friendly software compatible with our 
IBM 3033 system that facilitates the data collection 
and processing in support of the foregoing. 

The methods designed herein request proximity data or 
Similarity/dissimilarity data on pairwise combinations of 
professors in the Operations Research department. Also, the 
respondents are reaguested to provide ratings on several 
"bipolar scales't of suggested instructor characteristics. 
The methods for discovering the dimensions or relationships 
that appear to characterize the professors utilize several 
St aitea st icad LOqls wine Luds ng Multidimensional scaling, 
regression analysis, factor analysis and cluster analysis. 
The data used in the analysis comes from those students in 
the Operations Research curriculum graduating in March of 
[shes 

An interactive computer survey 1S designed to query the 
students on their perceptions of the teaching effectiveress 
Ch etheniImsreructors. Thus, data 1S input by the students 
during a twenty minute session on the 3278 terminal. 
Initiaily the student links to the sortware, and when 


finished, transmits his responses to a central file. 


Be SUBUECT 


The subject of this thesis deals with quantifying 
perceftions. Specifically, we would like to obtain an 
explanation as to why students perceive instructors as being 
Similar or dissimilar and discover the dynamic factors that 
a particular class uses to discriminate among instructors. 
The multidimensional scaling technique uses the information 
from the survey to create a ‘multidimensional map' of 
points. Each point represents the coordinate position of 
the objects under investigation, in our case, professors. 
Once this spatial plot 1s produced, it remains for the 
researcher to discover those factors that appear to cause 
the structural relationships. 

Multiple regression analysis and factor analysis are two 
technigues commonly used to describe linear relationships 
among dependent and independent variables. Each method is 
given consideration here in attempting to interpret the 
Spatial plot produced by the multidimensional scaling 
program, KYST [Ref. 1}. Additionally, cluster analysis is 
used to group the professors into disjoint clusters. This 
Cluster information is presented to the students during exit 
interviews to help guide the researcher in his attempt to 


find the underlying relationships. 


C. SCOPE 


This thesis 1s presented in four major chapters 
excluding the introduction. The second chapter describes in 
detail the background of multidimensional scaling methods. 
A brief account of regression analysis, factor analysis and 
Cluster analysis is given as well. The tkird chapter speci- 
fies the means by which the data was collected in the inter- 
active survey. Chapter iV emphasizes the analysis of the 
data. Finally, Chapter V provides a summary of the salient 


points determined in the study. 
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D. A BRIEF SUMMARY 


A four dimensicnal interpretation was emphasized in 
describing the data obtained from the computer survey and 
the SOF forms. These four factors included, 1) a student- 
instructor interaction effect; 2) the degree to which a 
professor was perceived as being organized or prepared for 
class; _ 3) a combined effect of grading policy, effort 
required outside class and pace of the course; and 4) a 
composite effect combining class size and the degree to 
which a course relied upon prereguiSsites. A high correla- 
tion appears to exist among those bipolar scales used in the 
current SOF form. Further investigation nopefully will lead 
to discovering other factors that will help describe teacher 
performance. The results obtained in this study are not 
meant to be predictive but explanatory. The value associ- 
ated with an instructor for each characteristic tmav be 
regarded as his score on that dimension. TAUS, Panieings “oe 


instructors by characteristics are possible. 


It. BACKGROUND 


A. WHAT IS MULTIDIMENSIONAL SCALING? 


Multidimensional scaling involves the problen of 
depicting n points in multidimensional space such that the 
interpoint distances correspond in some manner to measured 
proximity data [Beko 2: ps Sige The proximity data can be 
similarities, dissimilarities, correlation coefficients or 
any other measure of association as perceived by a_ set of 
judges participating im an experinent. Multidimensional 
scaling techniques attempt to produce the structure or 


interrelationships among the n objects by assuming a direct 


correspondence between the measured proximity data, Or 
dissimilarity data oe fr’ im our case, “and® the™interrpormt 
Gistances di. - Several choices of multidimensional scaling 


exist, the difference being the assumed relationship between 
Fa 
scaling (MDS) will be a spatial map that displays tne asso- 


and dj : The ultimate product of multidimensional 


Clation between the n objects under investigation. MDS has 
found considerable application in the social sciences, 
particularly in the realms Gf psychology, sociology, 
economics and education. 

A significant point worth pursuing is this idea of 
correspondence between the proximity data $.5 , and the 
distance data d.; , A fairly simple method of analyzing a 
possible relationship would be to observe a scatter plot 
Similar to the one given in Figure 2.1. The vertical Y 
axis of the scatter flot contains the measured dissinilari- 
ties dir while the horizontal X axis shows the corresponding 
distances, Gis , computed from the derived set of character- 


istic vectors. 
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Figure 2.1 Scatter Plot of Distance vs. Dissimilarity Data 


le 


It can be seen that an ascending pattern 1S created between 
the distances and the dissimilarities. One might even 
suggest that the relationship is linear, and could be 
described by an equation OE the form at Exe 
Multidimensional scaling methods that use a formula to 
describe distance as a function of dissimilarities are known 
as metric HDS. Metric MDS uses the numerical properties of 
the proximity data to determine distances. Another means by 
which distances are created from dissimilarity data without 
using the numerical properties of the dissimilarity data is 
knowh aS nonmetric MDS. Nonmetric MDS relies totally upon 
the rank ordering cf the dissimilarities to produce the 
distance data [Ref. 3: p. 22}. One would normally expect 
small dissimilarities to correspond to small distances and 
large dissimilarities to correspond to large distances. 
Although this relationship is rarely a perfect one, the rank 
orderings of the dissimilarity data is usually enough to 
create a good fit. Shepard and Kruskal [Ref. 2: p. 2], have 
done a substantial amcunt of work in nonmetric MDS and this 
particular method will be followed in this study. 

A review of some notation might be appropriate at this 
point. The n objects about which the investigator is trying 
to ascertain some fundamental relationship, can really be 
any sét of stimuli. For example, one might be interested in 
discerning the perceived distances between political candi- 
dates with hoves of discovering what issues or dimensions 
really set them apart from each other in the minds of the 
judges. Or, one might be interested in discovering the 
perceived distances Fketween countries in order to explain 
how Some countries might react in some political or econonic 
Sreuatlons. For this study, the n objects are the 
professors in the Operations Research department and we are 
interested in determining what factors cause the professors 


to be similar or dissimilar in the Minds ofvae £1 teamectreor 


VW 


judges. Faces phedkessormcanmebesthbought of «as @ podrt in 
multidimensional space. If it takes t dimensions to accu- 
rately describe this multidimensional space, then the coor- 


dinate describing professor x would be 
geeks eee eX, )- 


For h professors the coordinate system that results iooks 
like 


goo eX. roo eX) 


7 ee peek pee eX) - 


Thus, the entire space contains a configuration of n points 
each of t components. 

Each judge or student is asked to complete a survey that 
requires him to provide a value corresponding to how sSinilar 
or dissimilar he perceives each pairwise combination of 
professors to be. The scale used in the survey associates 
the value 1 with the meaning very Similar and the value of 9 
With the meaning very dissimilar. In order to determine 
rather than impose the relationships between professors, the 
Characteristics on which the professors are scored are not 
specified at this time [Ref. 3: eee 2) ile The notation for 
dissimilarity data in this study is dy, - This value repre- 
sents the perceived dissimilarity between professor i and 
proressor j. The end result of the data collection 1s 4 
Mion. OL eos ili babaty values Lor n stimuli that looks like 


the fcllowing, 


(pn 
a. 


‘ mi : : a! 


i 


In this study the dissimilarity values d,, and d-. were 
assumed to be the same (symmetric matrix), and the values 4 
, were ignored. This results in a partial matrix with upper 
triangular or lower triangular form not including the main 
diagonal. The total possible pairwise) comparisons ape tne 
8) ONY) 0 

The actual distance between professors 1 andj, denoted 
dq‘ , as calculated as a euclidian distance in the following 
manner for t dimensions 





like the similarity/dissimilarity matrix, the end result is 


= 


a matrix of distances such that di; =d., and dd @a10hwe ieSoreat 
out here that although we have specified the euclidean 
distance as the method by which distances are computed in 
the computer algorithaga, it is possible to substitute non- 


euclidean distances of tae forn 
me i | 
d (x,y)= Pale £ OE sole 
534 : 


In the field of mathematics, these distances are known as 
Minkowski r-metrics and are true distances in the sense that 


they satisfy the triangle inequality 
d-(x,Z)S d,(x,y)+d-~ (y,2Z) - 


Euclidean distances and Minkowski r-metrics share many prop- 
erties, however, they do differ when it comes to rotating 
the solution. When rotations are involved, any rigid rota- 
tion leaves euclidean distances unchanged. The only rigid 
rotations that leave non-euclidean distances unchanged are 
rotations that transform all permutations of coordinate axes 
into coordinate axes [Ref. 2: pp. 22-23]. 

Finally, the number of dimensions used to output the 
final mapping is not restricted in any way mathematically 


other than NT<N (N-1) /2. However, any visual interpretation 
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in greater than three dimensions becomes extremely difficult 
for anyone trying to discover the meaning of the grcufpingys. 
The stress, or goodness-of-fit neasure, which is descrited 
in the next section, is usually expected to reveal the best 


dimension for analysis. 


B. GOODNESS OF FIT: THE STRESS 


A performance indicator for each choice of t is needed. 
The customarily used function is known as the stress. 
Stress incorporates a fitting technique that measures the 
degree of nonmonotonicity between the dissimilarities §f., , 
and the distances d : tf a Contiguratiommeo: © points 
existed such that a perfect monotone relationship prevailed 
between the dissimilarity data and the distance data, then a 
perfect match would cccur and the stress would be zero for 
that particular dimension size and all dimensions greater. 
Unfortunately, zero stress rarely if ever occurs naturaily 
in data samples. The best choice then is to determine that 
configuration of points which minimizes the stress for each 
Choice of dimension, t. The method used to determine stress 
uses least squares monotonic regression, suitably normalized 
to produce a non-dimensional vaiue that indicates yoodness 
Orel Peers ep. 2-35 42 The lower the value of stress, 
them ctetter the fit. Kruskal [~RefsS 2: p. 3], has Deen abie 
to associate a verbal description with some specific values 
of stress Eased upon his experience. These values are indi- 
Cawed in Table 1 . In almost every instance, by increasing 
the number of dimensicns that describe the data, the value 
of the stress decreases. However, one generally anticirates 
that the amount of benefit associated with incrementing the 
dimensionality is marginaliy insignificant. in other words, 
a tradeoff exists between keeping the output ina lower 
dimension to make interpretation easier, if the stress 


improves only slightly. 
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TABLE 1 


Table of Stress Values According to Kruskal 
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Figure 2.2 illustrates an example of what we mean by a 
non-perfect match between the dissimilarity and the distance 
datas Here, the deviations measured along the horizontal 
distance axis di , between the starred coordinates and the 
circular coordinates, indicate the degree to which this 
configuration does not meet the monotonicity reguirement. 
The values a: are defined to be those numbers measured from 
the horizontal xX axis that minimaze stress subject tote 
constraint of MOnotenici ly Sp reto2- poco. 


C. DIMENSIONAL REQUIREMENTS 


The choice of how many dimensions are reguired to 
completely specify the output from the multidimensional 


scaling program is certainly NOt Iftiitively on7 ious, ee ere 
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Figure 2.2 The Degree to which the Data do 
not Satisy the Monotonicity Requirement. 
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are methods, however, which help to indicate why one choice 
of dimensionality would be more appropriate than others. 
The most obvious of these is to compare how stress, the 
goodness-of-fit measure, improves as a function of dimen- 
sion. One way to do this is to compute the best configura- 
tion for several dimensions and create a plot of stress vs. 
dimension to visually compare the results. It should be 
pointed out that the more dimensions one uses to explain or 
interpret the data, the lower the stress becomes. When the 
number of dimensions, t, exceeds the number of objects pinus 
one, n-1, the stress will always be zero [Ref. 2: p.j 16]. 
What one wants to lock for is that dimension above which the 
stress improves only slightly. It the @ddta is “good, a 
noticeable eibow will show up in the plot to indicate the 
appropriate dimension. We were not so fortunate in our 
present study, Figure 2.3 illustrates a stress versus 
dimension plot. 

Probably more than anything else, interpretability 
should te considered a key criterion to use in selecting the 
appropriate dimension for analysis. If it is possible to 
interpret the results of an MDS configuration in two dimen- 
Sions more readily than in say three dimensions, even though 
the stress 1S lower in three dimensions, one should consider 
employing the two dimensional interpretation. A final 
criterion suggested by Kruskal [Ref. 2: p. 16], depends upon 
the accuracy of the data. If an independent estimate exists 
to corroborate error free or near error free data, then one 
is allowed to extract more dimensions than one would under 
more error prone conditions. When all is said and done, the 
choice of dimensionality rests largely upon the experience 
of the experimenter. 
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Figure 2.3 
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D. THE MULTIDIMENSIONAL SCALING ALGORITHM 


Since the mathematical technigue for determining the 
optimal configuration, and therefore optimal stress, is 
somewhat complicated, only a brief description or what is 
considered necessary will be described here. Suppose that t 
dimensions are selected to describe a configuration of n 


points. Then 
(X,, gece edie eee eX. gees XQ 


can be used to describe a particular configuration in multi- 


dimensional space. For this particular “contre ueadt sen, ea 
specific value of stress exists. The overall objective is 
to make the stress value as small as possible. This “turns 


out to be a minimization problem of multiple variables and 
is handled by the method of steepest descent. Specificaliy, 
the algorithm begins at an arbitrary configuration and 
attempts to improve itself by moving in the direction that 
improves or minimizes the stress value quickest. The direc- 
tion of movement is known as the negative gradient and can 


be evaluated from the partial derivarives of the function 
S=f (Xi, 7 * 2 eae ee 

fRef. 4: pj 118]. For example, 
(-9S/ X\ gon04 G07 Xyeey eee oy ee 


is the negative gradient. Once the configuration reaches 
the point at which it can no longer improve in a particular 


direction, the new negative gradient is calculated and the 


process continues. Finally, when a configuration can no 
longer proceed in any direction with improvement, it has 
reached a iocal minimum. Hopefully, the local minimum is 


also the global minimun, but this is not necessarily true 
all the tine. 
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It is not always possible to deternine if the iocal 
Minimum is aliso the global mininuag, but some technigues 
exist to help verify that this iS so. For exampie, starcting 
the Minimization precess from several different initial 
configurations and ccmparing the final solutions will indi- 
cate if the same lccal minimum is achieved. The final 
configuration with the best stress value is nost likely the 
global minimum. There is nothing to guarantee that one will 
always achieve the global Minimum, but this is a common 
pee> lem typzeal of men-lineareoptimizationm problems. Im any 
case, the configuration is only useful if in the end it 
makes sense and gives insight to the experimenter [Ref. 4: 
pee ti9j. 


E. MULTIPLE LINEAR REGRESSION 


Once tne multidimesional scaling algorithm computes the 
configuration with the lowest stress, the researcher would 
like to determine the specific dimensions that underlie the 
data structure. One way to do this 1s to assume that a 
linear relationship exists between a dependent variable and 
several independent variables. Many linear regression 
models assume that the proportion of expiained variation cf 
the dependent variable is the sum of additive effects of 
Statistically Significant independent variables [Ref. 5: p. 
54]. Other regression models allow for interactive effects 
between independent variables. The dependent variable is 
said to be regressed over the independent variables. The 
result of this regression process is the coefficient o£ 
multiple determination, R2. The value oft R2 indicates the 
amount of variation in the dependent variable explained by 
the independent variables. The value of R@ ranges between 
O and 1. The naigher the value of R2 ,1.e., the closer it is 


ee) the better the model explains or predicts’ the 
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Gependent variable. The closer the value of R2 is to 0, the 
more likely it 1S that the model iS inappropriate jam 
accounting for the variation in the dependent variable. The 
reasons for a low value of R?2 are several. For one thing, 
the relationship under investigation may not be a_ linear 
one. If this is the case, linear regression modelling 1s no 
longer a satisfactory method to use to describe the rela- 
tionship. However, given that the relationship is truly a 
linear relationship, the reason for a low value of R? could 
be the result of specification error. What is meant by this 
is that the dependent variable for which an explanation is 
sought, is being explained by an inappropriate set or an 
insufficient number of independent variables. 

The artwork in regression modelling as well as in alter- 
native statistical modelling methods comes from being able 
to sugyest (or divine) the correct explanatory variabies. 
In €his  partteulan srudy- the explanatory variables used 
came from current student opinion forms and suggestions fron 


previous students in the Operations Research curriculum. 


Fe. FACTOR ANALYSIS 


AS mentioned before, factor analysis attempts to repre- 
sent a set of observed variables in terms of a smaller set 
of hypotheticai variables. The hypothetical variables are 
cnosen to account for the covariation among the originally 
observed variables. The number of common factors present 
among the observed variables can be estimated from the rank 
or the adjusted correlation matrix. 

Difficulties in factor analysis arise when the factor 
loadings are not known and have to be estimated from the 
covariance or correlation matrix. The problem is that given 
the correlation matrix for tne observed variables is known, 


any one of many cavsal structures could have produced it 
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Pret. 665 p. 38). The three types of problems that result 
from uncertainties about the covariance matrix and the 
underlying causal structure are; 1) specific Covarianee 
matrices can be created by factor models with the same 
number of common factors but with different factor loadings; 
2) specific covariance matrices can be created by factor 
models with different number of common factors; 3) certain 
covariance matrices can be created by factor analytic causal 
models as well as non-factor analytic causal models [{Ref. 6: 
Deweao |. 

Two assumptions ccmmonly made in factor analysis are the 
postulate of factorial causation and the postulate of parsi- 
mony. Basically, the postulate of factorial causation 
reguires that the researcher show that the originally 
observed variables are a linear combination of some causai 
variables [Ref. 6: pe 43). The postulate of parsimony 
allows the researcher to assume a factor nodel with a 
smaller number of factors, given that two factor models with 
different numbers of factors have the same covariance 
structure [Ref. 6: p. 44]. 


G. CLUSTER ANALYSIS 


A final topic used to assist in this study is cluster 
analysis. Cluster analysis is aclass of techniques that 
morcallvy place sobayects into. groups ofyclusters. suggested Dy 
the data such that objects ina cluster tend to be similar 
to objects in the same cluster and dissimilar to objects in 
other clusters [Ref. 7]. The type of ciuster analysis 
followed in this study is disjoint cluster analysis whereby 
objects may belong to one and only one cluster as opposed to 
hierarchical cluster analysis where one cluster night be 


contained within another. 
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IIf. METHODOLOGY 


A. LAYING THE FOUNDATION 


The first step in this study to understand perceptions 
was to decide upon aocontrol group of students from whon 
data would be obtained. In this particular instance, the 
judges were twenty-three Operations Research students from 
the section that graduates in March of 1985. The twenty- 
three students included two foreign nationals, two Marine 
Corps officers, four women naval officers and fifteen nale 
naval officers all with various educational and career 
back grounds. 

The information that was to be gleaned from these 
students was simply this: How do professors in the 
Operations Research department at the Naval Postgraduate 
School differ, and in what ways are they similar in teaching 
styles and methods? There were several methods available to 
obtain this informaticn. Most simply, a hand written survey 
with many guestions could have been developed and handed to 
each student to complete and return. The decision was made 
to handle the data collection vila a computer interactive 
survey in order to minimize the logistics and data manipula- 
tion froblems associated with a hand written survey. The 
logistics problem was not a great problem per se, it just 
meant that a hand written survey had to be distributed and 
collected with plenty of opportunity for tne surveys to 
become misplaced. From a data manipulation point of view, 
the data collected from a hand written survey would almost 
certainly have to be recorded on computer in order to be 
able to use the results easily and guickiy. As a result, 


the decision was made to attempt to collect the information 
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about the professors by having the students compiete an 
interactive computer survey at a Naval Postgraduate School 
computer terminal with display screen. Aithough the data 
collection in this manner wasS not entirely free from its own 
brand of difficulties, it eliminated the requirement for any 
paperwork to be handed out or returned. All the results 
were automatically sent to the researchers computer storage 
at the completion of the survey. Also, the data was ina 
ready-to-use state by being contained on computer disk, and 
hence the need to transcribe results disappeared. 
Unfortunately, there are numerous contingencies for 
which a computer programmer must plan in order to designa 
computer survey that is simple, thorough, and user-friendly. 
The interactive computer survey designed for this study 
served its purpose well. An example of this progran, 


Teuetensinetfoktman, is mrcludcded in Appendix D. 


B. THE COMPUTER SURVEY 


Several important requirements that had to be addressed 
in designing the interactive survey included simplicity, 
thoroughness and brevity. It was considered important to 
collect accurate responses and to limit the workload as nuch 
aS possibile. Each judge respondec to ore hundred-seventy 
items on the average. Even with this many questions to 
answer, each student appeared to have completed the survey 
on the order of twenty minutes. oven still, more work 
should be done to reduce the amount of informaticn askcc of 
each judge. 

The nature or the computer survey progressed as follows. 
Each judge was required to link to the researcners computer 
disk, and once accessed, issue a command to run the execu- 
tive program that drove the survey. Immediately, a panei of 


the names of the professors in the Operations Research 
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department would appear, and the judge would be asked to 
indicate those professors from whom he had taken a course. 
After selecting his own subset of professors, the judge was 
asked to rate each fairwise combination of professors int 
terms of similarity or dissimilarity of teaching style. Jule 
the judge had observed n professors, this meant that a judge 
would have to respond to at least n(n-1)/2 prompts for this 
proximity data. For this study, the sixteen professors are 
identified by using the letters A thru P. The scale 
presented to the judge ranged from a value of 1 meaning very 
similar to a value of 9 meaning very dissimiiar. The judges 
were not limited to integer responses, Lut real responses 
were restricted to one decimal place. It was assumed that 
the proximity scaie was an interval scale meaning the 
distance in Similarity or dissimilarity values between say 
values 2 and 3 was equal to the distance in dissimilarity 
between values 8 and 9. 

Once all the proximity data had been collected, the 
judges were then asked to score the professors with respect 
to several bipolar scales. In the case where students had 
taken a professor for more than one course, the student was 
instructed to respond to the bipolar scales based on the 
last course taught by the professor. The scales used in the 
survey appear in Table 2 and were suggested by previous 
Operations Pesearch students. The tkipolar scaies also 
ranged in value from 1 to 9. Once each judge completed the 
survey, his or her results were automatically sent to the 
researchers computer disk for subsequent evaluation. 

In addition to the data collected from the computer 
interactive survey, information gathered fFrom student 
Opinion forms (SOF's) were used to try to interpret charac- 
teristics of the different professors. Only those student 
Opinion forms from those classes taught by the professors 


selected in the interactive survey were considered. The 
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TABLE 2 


Bipolar Scales Used in the Computer Survey 
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bipolar scales used in current SOF forms appear in Table 3. 
One problem encountered in the evaluation of these foras 
feet Wien Malntalning the purity of the control group. For 
the most part, the SOF's were completed by the twenty-three 
SsaldentS a2mearhe control groups dowever, there were sone 
instances where other students either from other Operations 
Research sections or fron other curriculums were includéd in 
the evaluation process. The information coliected from the 
SOF's represented different scales upon which to measure 
teacher performance. The SOF data was originally converted 
to an interval scaie on a range different from the scales 


used in the interactive Survey. Because linear 
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TABLE 3 


Bipolar Scales Used in the SOF Forms 
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transformations of the form atbx are allowed on interval 
scales, the scale used for the SOF's was changed to match 
the scale used in the interactive survey. In the case of 


each set of scales, a separate regression analysis and 


28 


factor analysis was done. One kev parameter we were 
interested in measuring was the correlation between how the 
judges evaluated the professors overall performance at 
different times in the curriculum. SOF data was only avail- 
able through the end of the Summer quarter of 1934 (the end 
of the judge's sixth quarter). Since the judges completed 
the computer survey at the beginning of the eigth guarter, 
only those professors taken through the end of the Sixta 
quarter were evaluated. Tais meant that a few professors 


were not included in the final evaluation. 


C. OUTPUT FROA MDS 


The proximity or diSsimiiarity data obtained from the 
interactive survey was used as a direct input toa nmultidi- 
Mensional scaling algorithm KYST, partially developed cy 
Kruskal [Ref. 1: p.j 1}. The proximity input values for this 
study appear in Appendix a. The output from the multidimen- 
sional scaling program includes visual plots or spatial maps 
depicting the professors positions in multidimenSionai space 
projected down to two dimensions for visual display. Also 
included as output are coordinates for each professor in 
multidimensional space. The task that remained was to 
define the variables that best explained the location o£ 
each professor. 

One method avaiiable for determining what characteris- 
tics explain the orientation of the professors from the 
multidimensional scaling output is multiple linear regres- 
sion. The median value from each bipolar scale is regressea 
over the coordinate fositions of each professor. From the 
stress versus dimension curve it was decided to concentrate 
on a four dimensional interpretation, although three and 
five dimensional interpretations were considered also. the 


vaiues of stress, the goodness-of-fit function, turned out 
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to be 0.249 Zor f£otz) dimensions) anayyo2274) corer 
dimensions. Neither of these values indicate a very good 
fit according to Kruskals' own personal experience. This 
large value of stress was anearly indication that this 
linear model might not be appropriate in explaining teacher 
verformance. | 

Figure 3.1 shows the resulting regression weightS or 
direction cosines corresponding to each multiple correlation 
for the four dimensional solution. The direction cosines 
are regression weights normalized so that their sun of 
squares eguals 1.0 for every scale [Ref. 3: p. 37]. EOuE 
example, when regression weights of 0.4178, 0.8959, -0.1508, 
and -0.0026 are given to dimensions 1,2,3 and 4% respec- 
tively, the multiple correlation between the resulting coor- 
dinate positions and the respective bipolar scale is 0.5931. 

A bipolar scale will provide a good interpretation of a 
dimension when its multiple correlation coefficient is high. 
A value above .90 is desired. The values achieved in this 
study were iow. Values of R@ close to 0.5 were typical. 
Also a requirement for good dimensionai interpretation is a 
high regression weight on the dimension it most nearly 
explains. The results obtained from the bipolar scaies used 


in this study are examined in the next chapter. 


De. OUTPUT FROM FACTOR ANALYSIS 


Having looked at regression modelling as an approach to 
interpreting dimensions, factor analysis was also considered 
a possible means of identifying linear factors that would 
help describe teacher performance from the data sample. The 
statistical analysis rpackage, SAS, was used to generate two 
separate factor analysis outputs. One factor analysis was 
completed for the bipolar scales associated with the 
computer survey and another was done on the additional 


bipolar scales from the SOF forms. 
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Pagiine 3 | Regression Weights for the 


Four Diaensional Solution. 
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Initially, a correlation matrix was produced from the 
raw input data. Additionally, the common factors and the 
factor loadings for each observed variable or bipolar scale 
was produced. Orthogonal rotations were effected to produce 
Simple structure. A total of two common factors were 
created from the bifelar scales of the interactive survey, 
and one common factor was produced from the SOF data. The 
specifics of the factor analysis output are discussed in the 


next chapter. 


E. OUTPOT FROM CLUSTER ANALYSIS 


As a final measure, a disjoint cluster analysis was 
performed on the MDS data. The clustering routine, 
Fastclus, waS available from the statistical analysis 
package, SAS. The number of clusters into which the grcunp 
Of instructors were subdivided was specified by the 
researcher to range from 2 to 6. Membership in a particular 
cluster was determined based upon the distance from each 
professors position to the mean value of the cluster. The 
output from the cluster analysis included identification of 


the ciuster to which each professor belonged. 
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IV. DATA ANALYSIS 


> a > _—————-a ee oe 


Aw. INTENT OF THE ANALYSIS 


The scope of this chapter will be to analyze the results 
obtained from the completed interactive survey in the 
context of multiple linear regression and stepwise regres- 
sion, factor anaiysis and cluster analysis. ae 10 nal 
information gathered from student opinion forms (SOFs), will 
be evaluated in so far as what characteristicS or bipoiar 
ScaleS appear to have been most important in describing 
students perceptions of teacher performance. 

An issue that requires explanation before the analysis 
begins concerns a vital assumption made dealing with the 
scale of tne data. Specifically, can an arkitrariiy chosen 
numerical scale with fixed upper and lower bounds’ enactle 
correct statistical inference from the data sample? In our 
Study, we provided the judges with a numerical scale ranging 
in value from 1 to 9. The judges were allowed to rate each 
professor in every category with respect to this scale. In 
essence we are imposing an interval scale, with equally 
spaced intervals. Let's consider a judge's response to the 
Gestion of grading policy. It may be that the judges can 
rate the different objects, in our case professors, at best 
on an ordinal scale. The judge may be able to say that 
professor Ais a harder grader than is professor B, but not 
how much harder. We are aSSuming that the judges, when 
responding to these bipolar scales, realize that we are 
anfering an interval scale on their responses. Fe assume 
that when they complete tne survey, that they realize that 
each integer value on the scale divides the Scaie into ejual 


intervals. We assume that a judge will rate each professor 
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knowing that the distance between say a score of 3 ard 4 is 
equai to the distance between a score of 5 and 6. We find 
ourselves doing this in order to follow the example of 
Kruskal as closely as possible and because the statistical 
methods used to evaluate the data require at least an 
interval scale. Given that we are assuming an interval 
scale, we feel comfortable in transforming the data with any 
linear transformation of the form a + bx. The majer point 
being made here is that the researcher should keep in mind 
the scale of the data when interpreting the significance of 


the output. 


B. WHAT DID WE LEARN FROM MDS? 


As mentioned in Chapter II, multidimensional sScaiiny 
attempts to determine the structural relationships between n 
objects from a matrix of halftatrix ©£ proximity daar A 
Major result of the process is the spatial mapping cf tne rn 
objects, usually projected down to the planes of each pair 
of dimensions for easy visual interpretation. The final 
configuration of points represents the best fit of then 
objects according to the stress criterion. Table 4 contains 
the final configuraticns for all sixteen professors in four 
dimensions. 

In order to determine which dimensionality scheme best 
Suggests the characteristics that set the professors apart 
from one another, it helps to review the spatial maps 
created from the data. Appendices 3B and C contain the 
Spatial plots of the sixteen professors for four and five 
dimensional solutions. Each pairwise combination of axes is 
plotted against each other. This spatial orientation can 
sometimes suggest from mere inspection those characteristics 
that cause some professors to be more alike than others. 


Fowever, we do not usually rely upon visual inspection 
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Final Configuration for the 16 Professors 
in 4 Dimensions 
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alone. Very often a plot indicating how Stress Improves as 
a functicn of dimension gives the dimensional interpreta- 
ten. Figure 4.1 is such a plot for the data collected in 
this study. It can be seen that a one dimensional interpre- 
tation yields a very high stress value of 0.463. This would 
suggest that a one dimensional interpretation alone would be 
inappropriate. We would like a noticeable elbow to occur in 
the stress vs. dimension plot, for this normally indicates 
the most suitable level of interpretation. We observe that 
this plot does not exhibit the noticeable eibow. Instead, 
the slope of the curve decreases gradually and we are ieft 
to look for other means to help determine dimensionality. 
Kruskal and Wish offer an alternate method or rule of 
thumb in choosing dimensionality [Ref. 3 p. 34}. A oe 
suggest that the number of stimulus objects thinus one, in 
our case fifteen, should exceed four times the dimension 
chosen for interpretation. This would offer a choice of 
dimension no greater than 3.75. They caveat this statement 
by saying that this rule has only been found to hold for 
three dimensions, and that further study is needed to see if 
it 1S approj,riate for higher dimensions as well. we chose 
to emphasize a four dimensional interpretation, however a 
three and five dimensional interpretation were considered 


also. 


C. THE USE OF REGRESSION 


Once the choice of dimensionality had been made, the 
next task was to determine which characteristics represented 
those dimensions best. LO sdomenkcr we deciced, as do 
Kruskal and others [Ref. 3: pp. 35-36], to use multiple 
linear regression as a means of clarifying this issue. 
Again, we reiterate that a separate regression analysis was 


conducted for the scales used in the computer survey as well 
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as the scales obtained from the SOF forms. Specifically, we 
regressed the median value of each bipolar scale over the 
coordinates of each professor, and recorded a multiple 
correlation coeificient. If the value of the multifle 
correlaticn coefficient was high, i.e., close to 1, then we 
felt inclined to believe that this scale was important in 
distinguishing professors. If the value of the multiple 
correlation coefficient was low, i.e., close to 0, then that 
particular scale was not perceived as being important. 
Tables 5, 6 and 7 illustrate the two sets of bipolar 
scales used in the study, the corresponding normalized 
regression weights or direction cosines, and the gultiple 
correlation coefficients associated with each scale for 
dimensions 3, 4and 5. It can be seen that for the nost 
part, the multiple correlation coefficients hover near 0.5 
and typically increase as the the dimension increases. 
Figure 4.2 depicts the changes in the amuitiple correlation 
coefficient for each bipolar scale as the dimension for 
interpretation changes. In the table, bipolar scales 1 thru 
6 refer to the scales from the computer survey and bipolar 
scales 7 thru 17 refer to those scales from the SOF forn. 
Since none of the scales exhibited truiy high correla- 
tion coefficients, we chose those that had the hignest 
values and used them to interpret the dimensions in our 
study. The regression analysis on the computer survey 
scales suggested that these scales were inappropriate indi- 
cators for this set of judges. For the four dinensional 
analysis, we see that the bipolar scales 'theoretical vs. 
applied course' and ‘course relied upon prerequisites" have 
the highest correlation coefficients for witmat Sacelp of 
scales. Specifically, the correlation coefficients are .510 
and .501 respectively. The scale ‘applied vs. theoretical 
course’ loads heavily on dimension 2 with a normalized 


regression weight of -0.8367. Thus, it would seem that the 
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ression Weights and Multiple Correlation. 
Coetticncnes for the Bipolar Scaies in Dimensions 
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fact that a course was perceived as being theoretical or 
applied most closely describes the erfect of the professors 
ultimate positioning with respect to dimension 2. Likewise, 
the scale ‘course relied upon prereguisites' loads heavily 
on dimension 4 with a regression weight of -0.7270. Paws 
appears to suggest that dimension 4 is most nearly explained 
by how the judges ferceived how each course relied upon 
other prerequisite courses. The regression weights have a 
geometrical interpretation as the cosine of the angle 
between the dimension upon which it loads, and the associ- 
ated scale. Unfortunately, the two scales with the next 
highest multiple correlation coefficients, ‘pace of course! 
and ‘grading policy', ioad heavily upon dimensions 2 and 4 
as well. They probably contribute to the explanation of 
those dimensions aiso. Not one of the scales with a high 
multiple correlation coefficient loads heavily upon dimen- 
Sions 1 or 3. These dimensions are left unexplained and nex 
scales are needed to determine an appropriate explanation. 

A similar analysis was conducted for the bipolar scales 
used in the SOF forms. Here the scales with the highest 
multiple correlation coefficients were 1) instructors objec- 
tives made clear; 2) instructor cared about student progress 
and did his share in helping to learn; 3) course organiza- 
tion and 4) time in class spent effectively. Again, the 
same problem occurred here as in the analysis of the 
previous set of scales. Almost all those scales with the 
highest multiple correlation coefficients load heavily on 
only two of the four dimensions, NMhameiy dimension 1 and 2. 
Dimension 1 appears to convey information about how each 
Frofessor cares or interacts with his students on a personal 
level. The bipolar scales with high multiple correlation 
coefficients that load heaviiy on this dimension are, 1) 
instructor knows when students don't understand the 


Material; 2) difficult concepts were made understandable; 3) 
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felt free to ask questions; Wyinstructor made course a 
worthwhile learning experience and 5) instructor cared about 
student progress and did his share in helping to learn. 
Dimension 2 seems to describe information concerning how 
instructors are perceived as being organized or prepared for 
their course. The scales with high multiple correlation 
coefficients that load heavily upon this dimension are 1) 
course organization; 2)time in class spent effectively and 
3) instructors objectives are made clear. 

in addition to the bipolar scales used for the interac- 
tive survey andthe SOF forns, each judge was asked to 
provide a rating of overall teaching performance for each 
professor. The median values for each professor were 
obtained and are tabulated in Table 8 . Although we have 
kept the analysis of the interactive survey separate fron 
the SOF data, we were interested in the correlation between 
professor's overall performance judged on two separate 
occaisions. The first occaiSion occurred when the judge 
completed the SOF form for the course taught by the 
professor. The second occaiSion was in conjunction with the 
interactive survey. The correlation between ratings of 
overall] performance for the professors was .58, lower than 
expected. Suggestions as to why this correlation is low 
include the effect of time. Judges may be less able to 
evaluate a particular instructors overall performance as 
time goes by. Also, students may change their opinions of 
teachers overall performance after having seen a professor 
teach a variety of different ciasses. For example, a 
student who enjoys applied courses might rate a professor 
differently after having had him fora Stochastics Models 
course than he would for say an applied course like Test and 
Evaluation. in any event, we did expect the correlation on 


overall rerformance to be higher than it turned out to tLe. 
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Steowise regressicn was utilzed  in-.order to determine 
which scales or independent variables explained the greatest 
proportion of variance in teacher performance. The APL 
function STREG, was used to compute the order in which the 
independent variables were to enter into the regression 
model. The output from STREG is displayed in Table 9 for 
both the six scales asked in the computer survey as well as 
the scales from the SOF forms. The output indicates the 
order in which the variables entered the model, the coeffi- 
cients associated with each independent variable, the stan- 
dard error associated with each coefficient, the 
corresponding t-value and the proportion of the variation of 
the dependent variable accounted for By the independent 
variable. we see that the coefficient of multiple determi- 
Nation, R?2, is low for the six bipolar scales used in the 
computer survey. In all, these scales account for only 13.8 
percent of the variation in the dependent variable, overall 
instructor performance. However, the stepwise regression 
performed on the eleven bipolar scales from the SOF forms 
indicate a coefficient of multiple determination of .986. 
This set of scales explains more of the variation in the 
judges responses to overall instructor performance than does 
the previous set of scales, In partieovulanr,, "comprce worgans — 
zation' appears to have a significant impact on students 
perceftion of teacher performance since it accounts for over 
73 percent of the variation alone. 

One problem that concerned us dealt with the fact that 
for the SOF data, even with a value of R2 equal to .986, 
most of the independent variables were statistically insig- 
NLELCant. Even the scale ‘course organization’, which 
accounted for 73 percent of the variation in overall 
performance , proved to be statistically ainsignificant at 
the 0.05 level. A high value of R2 and statistically 


insignificant variables usually is symptomatic of high 
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Figure 4.2 Changes in the Value of the a. Correlation 
Coefficient as a Function of Dimension 
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multicollinearity. Multicollinearity simply means that one 
or more so Called independent variables are highly ccrre- 
jated with another independent variable or is a linear 
combination of a number of the independent variables. The 
problem with high multicollinearity is that the estinates 
for the regression coefficients become unreliakle from one 
sample to the next. Our confidence in our ability to deéter- 
Mine the efrect of an independent variable withers. To show 
that high multicollinearity exists, we need to regress each 
independent variable over all the other independent vari- 
ables to see if any are a linear combination of the others. 
Table 10 shows the values of R2 obtained by regressing each 
independent variable over the others for both nodels. 
Several options are available when confronted with high 
multicollinearity as we were in this study. One solution is 
to increase the sample size. This turned out not to bea 
useful alternative Since our Sample size was fixed. Ancther 
strategy 1S to combine several variables that are highly 
correlated into a single indicator as long as it makes 
sense. This is possible for several scales on the SOF forz 
which are highly correlated. A third alternative is_ to 
discard those variables which are linear combinations of the 
others and are the cause for the high muiticollinearity. 
After discarding the offending variables, anew regression 
Sequationuecan bewreated and a check for statistical Signiri- 


cance made anew. 


De. RESULTS OF FACTOR ANALYSIS 


Factor analysis supposes” that some common factors 
smaller in number than the originally observed variables, 
account for the covariation of the originally observed vari- 
ables. Factor analysis assumes a linear causal relationship 


Similar to linear regression analysis. Por £his study, tio 
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RESULTS FROM STEP wiSF REGRESSION CF BIPOLAR SCALES 
SURVEY ON [ENSTRUCTOR OVERALL PERFCRMANCE. 


OROER [(N #HI{ CH VARIABLE REGRESSICN STAAQARO 

ENTFREOQ THE MOOEL COEFFICLENT ERROR 
THE ORE T(C4L VSe APPLIEO 0.214 267 

EFFORT REOULREQ OUTSIDE 0.4835 0.647 
CLASS 

PACE OF COURSE 22392 3-225 

CCURSE RELIEO UPON PRE=- —-9 .483 0.844 
REQUISITES 

CLASS SIZE 9.028 0-€5S5 

24 OF [INSTRUCTOR Oveane opted eabrnanes 5 aY 

RESULTS FROM STEPW{SE REGRESSICN OCF BLPOLAR SCALE 

ON ([NSTRUCTOR OVERALL FERFORMAACE 

OROER IN WHICH VARIABLE REGRESSION ST4KQ04RO 

ENTEREQ THE MOQOEL COEFFICIENT ERROR 

COURSE ORGAN[ ZATION 1.070 Oe411 
INSTRUCTOR KNOWS *HEN 1 «234 04.224 
STUDENTS UNOQERSTANO 

FELT FREE TO 4SK QUES= —-1.819 0.339 
TLONS 

CONF LDOENCE IN ENSTRUCTCRS 2.062 0-040 
KNOSWLEOGE IN SUBJECT . 

INSTRUCTOR STIMULATED [N= -0.415 0.416 
TEFEST {N SUBJECT AREA 
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[INSTRUCTOR CAREO aAsOoOUT STU (eos 0.629 
OENT PROGRESS 4NO OfO HIS 
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@ORTHWHILE LEARNING 
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[INSTRUCTOR PREPARED FOR 0.679 0-400 
CLASS 

OLFFICULT CONCEPTS MADE 0.575 0-240 
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class S17 —— a ees25 | 
THEORETICAL VS. APPLIED 0.245 

| GRADING POLICY 0.951 ! 
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| DEPENDENT VARIABLE RESULTING R=SCUARED 
COURSE ORGANI ZATION 0.961 
TOME TN CLASS SPENT EFFECTIVELY 0.941 
| INSTRUCTOR KNOWS SHEN STUDENTS CON’ T 0.905 | 
UNDERSTAND MATERIAL 
OLFFICULT CONCEPTS MACE UNDERSTANDABLE 0.947 
CONFIDENCE IN INSTRUCTORS KNO#LEDGE IN 0.762 
SUBJECT 
| FELT FREE TO ASK QUESTICNS ! 0.9al 
INSTRUCTOR PREPARED FOR CLASS 0-963 
INSTRUCTORS OBJECTIVES “ACE CLEAR 0.848 
INSTRUCTOR MADE COURSE «ORTHanILE 0.986 | 
LEARNING EXPERIENCE | 
INST2UCTOR STIMULATED INTEREST IN 0.930 
SUBJECT AREA 
INSTRUCTOR CARED ABOUT STUDENT 0.980 
| PROGRESS ANO DID HIS SHARE IN | 
HELPING TO LEARN : 
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separate factor analyses were conducted, one or each set of 
bipolar scales. Each will be discussed separately. 

The factor analysis conducted on the six bipolar scales 
used in the computer survey yielded the results shown in 
Table 11. In addition to the correlation matrix, the unro- 
tated factor loadings and the orthogonally rotated factor 
loadings appear in the table. Variables with factor load- 
ings that are close numerically suggest a common interaction 
or measure. It seems as though two common factors are 
present in the six variables based upon the factor loadings. 
The first factor explains 44 percent of the variation in the 
data. This Factor seems to combine the affect of ‘grading 
polie yy, ‘effort required outside class', a var pac ce on 
course'. The second factor appears to be a composite effect 
of ‘class size’ and ‘course relied upon prerequisites'. fThe 
scale ‘class size' has a high positive factor loading 
whereas the scale ‘course relied upon prerequisites' hasa 
fairly high negative loading. This would seem to make sense 
Since those classes taken early in the curriculum tended to 
be large and the earlier courses did not usually require a 
Significant amount of prerequisite courses. 

The results of the factor analysis on the SOF data 
yielded the correlation matrix and factor loadings in Table 
12. ‘From this factor analysis, we see that oniy one common 
factor accounts for the variance in tke data. This is 
reasonable since most of the bipolar scales from the SOF 
forms are highly correlated. This factor could describe the 
student-professor interaction effect discussed earlier. 

Figures 4.3, 4.4, 4.5 illustrate boxplots of factor 
scores for each professor for all three factors. Factoren 
again is an indicator of the combined effect of ‘grading 
policy', ‘pace of course', and teffort required outside 
Cclass'. It might be condensed into a general workload index 


with a high factor score indicating those professors for 
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TABLE 11 


Table of Factor Analysis Results on Survey Data 
SCALE t= CLASS SIZE 
SCALE 2= APPLIED VS. THECRETIC4L COURSE 
SCALE 3= GRADING PNLICY 
| SCALE 4= PACS AF COURSE 
SCALE S= EFFONT REQUIRED OUISLIOE CLa4SS 
| SCALE 62 COURSE RELIED LPCN OKERECUISITES 
! CORRELATION MATA[«A 
SCALE 1 SCALE 2 SCALE 3 SCALE a SCALE 5 SC4lc 9 
SCALE 1 1.9000 -9.1327 9.14690 0.0647 0.0449 -0.2815 
SCALE 2 -9.1327 1.ocao0 Jes m0 0.2671 Oe rae | 9.27453 
| SCALE 3 0.1300 OIL LO 1.9000 0.5124 0.4104 0.3357 
SCALE 4 19.0647 7.2671 9.5124 b.0C00 0.5751 0.4924 
SCALE 5 0.0463 Onl oat 9.6104 0.57581 1.0090 0.5040 
SCALE $& -0.2615 9.2763 9.3357 0.4¢2¢ 0.5040 L.200u 
| FaCTQR PATTERN 
NO ROTATION ORTFOGNNAL 2QTAriON 
Factcr 1 Pecron 2 Pac ron | Pret CRY 2 
scacz 1 -0.021485 0.71925 0.24550 0.30614 
| SCALE 2 3.40519 -9.30547 0.37634 -9.43293 
SC4te 3 9.77491 e 22461 G22 25606 0.902412 
SCALE & 9.30453 0.11096 0.f0215 -9.12597 
SCALE S 9.82597 O.1S145 0.33442 -9.99443 
SCALE 6 0.72015 -9.41322 9.56949 -0.60420 
| EIGENVALVES OF THE CORRELATION ¥ATRIX: TOTAL =5.90N000 AVE 24GE =1-90000 
| l 2 3 a 5 4 
FIL GENVALUE 2.0344 tLe2ay7 O-.d3o6l 0.494923 O25 052 0.3233 
| OLFFO RENCE 1.4347 0.4156 0.5379 C.0932 0.0749 
PRCEIRTILON 0.4474 0.2032 0.1393 C.0831 0.0572 0.0547 
CUMULATIVE 0.4474 0.0557 0.7950 O.u731 0.9453 1.0000 
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| TABLE tz | 
| Table of Pactor Analysis Results on SOF Data | 
| SOFL=COURSE ORGANI ZATICN 
| SOF 2=TIME {tN CLASS | SeeNt (ere eer tveLy 
SOF 3S (INSTRUCTOR KNOWS «REN STLOENTS OONST UNCERSTANO MATERL AL 
SOFG=OLFFICULT CONCEPTS “ADE UNCERSTANOABLE | 
SOF S=CONF LOENCE INSTRUCTORS KNOWLECGE [tN SUSJECT | 
} SOF6=FELY FREE TO ASK QUESTIONS 
SUF7ZLNSTFRUCTCR PREPARFO FCR CLASS 
SOFB=(NSTRUCTURS OSJECTIVES “ADE CLEAR 
SOF9=INSTRUCTOR “AQE COURSE wsORTHWHILE LEARRING EXPERIENCE 
SOFLO=INSTRUCTOR STIMULATED INTEREST IN SUBJECT 4REA 
| SCFLLSZINSTRUCTOR CARED aBCur STLOENT PRQGRESS A4NO OLO HIS SHARE IN 
HELPING TO LEARN | 
CORRELATION MATRIX | 
SOFt SOF2 SOF3 SCFS SOFS SOF6 SCF7 SOFS SOF9 SOFI!O SOFtt 
| SOFt te900 04670 04660 04.68 0.4.42 04649 0.646 0-69 0.69 0.57 0.65 
SCF2 0.70 11.00 04662 02.67 0.43 “0.63 O00 .S9 (0. 70d.o9 0-60 | 
SOF3 04660 0.62 1.00 0.72 9.42 9.54 0.56 °° 0.52 0766 “O..57 Oer2 
| SCF4 0.63 0.87 0.72 t e900 0.4 3 O.35 0.62 Om a5 G.70 0.58 0.79 
SOFS 0.642 04243 04642 0443 1-00 04637 2435 0.35 0.50 0.44 0.46 | 
SOF6 0.49 9.43 0.54 0.655 0437 1.00 4.57 0.44 0.53 0.46 Q.6! 
SOF7 0.638 9-60 0458 04662 904648 0.57 1.00 0.57 0.63 0.55 0.66 
SOF83 04.669 9.59 04.652 04655 9635 0448 0.57 1.00 0.60 0.50 0.58 | 
SOF9 0.69 064670 9.60 0470 04.50 0.53 04.623 04.60 1.00 0.74 Oi. 70 
SOFIO 0.57 0.59 0.57 04.58 04.44 0.46 04.50 0.50 0.74 1t.00 0.59 
SOFLt 0.65 0.60 0.472 O4.70U 9.4.46 04.61 04.66 0-58 04.70 0.59 {00 
FACTOR PATTERN = EIGENVALUES OF THE CORRELATION MATRIX 
= TOTAL= 1t.o00G00 AVERAGE= 1.00000 
FACTOR = 
| = t 2 3 
SOF! 0.33906 = ELGENVALUE 6675680 072500 .57860.5924 
+ OLFFERENCE 6.0718 0.0464 0.0858 02-1046 
SOF 2 0.30779 = PROPCRIION 0.6179 0.0659 0.0017 0.0539 
] + CUMULATIVE 0.6179 0.6838 0.7455 0.3438 
l SOF3 0.d1009 s 
= = 6 7 
SOF« 0.84420 = ELGENVALLE 0.4 886 0.3862 0.3430 0.2861 
= OLFFERENCE O.1C21 0.0432 09.0568 0.0310 
SOFS 0.59563 ¢ PROPGRTION 0.9444 0.035t O03 12 0.0260 
= CUMULATIVE 0.8438 Ue38789 0.9101 0.9 361 | 
| SOF6 0.08709 + 
~~ 9 to tt 
SOF 7 0.40055 = ELGENVALUE 0.2552 0.2256 0.2124 
2 OLFFERENCE 0.0196 0.0252 
SOF2 0.74319 + PROPCRIION 0.02412 0.0214 0.9193 
| = CUMULATIVE 0.9593 0.9807 1.0000 
SOF9 O.a7031 cs 
| SOF 10 0.76053 = | 
SOF tt 0.384586 | 
a Gen ce a 
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whom the workload was a concern to the judges. Professors 
B, H, K, lL, and P all have high median factor scores with 
respect to this common factor. Factor 2 might represent a 
composite effect between the Scales ‘class size' and ‘course 
relied upon prereguisites'. Professors B, I and N had high 
factor scores with respect to this indicator. Paectccr @ 
encomfasses a general student-professor interaction effect. 
Most of the professors had the same spread of factor scores 
except for professor K. Here the judges exhibited a higher 


level of controversy for this characteristic. 


Es. RESULTS OF CLUSTER ANALYSIS 


A final means of looking at the groupings of professors 
focused on a cluster analysis. In this technigue, 
professors who were fperceived to be similar to one another 
were grouped together in the same cluster. Other professors 
were likewise grouped in other disjoint clusters so that no 
professor belonged to more than one cluster. The fprotessors 
were assigned to clusters based upon their interpoint 
distance from the cluster means. An initial cluster seed is 
selected and the iterative process continues until the 
observations become stable, 1.€., each observation settles 
down into a steady state ciuster. It remains for the 
researcher to decide upon the number of clusters that rest 
describe the groupings of professors. One strategy that was 
employed in determining the appropriate number of clusters 
resuited from interviews with the students. Tables 13 and 
14 display the clusters to which each professor belonged 


from the four and five dinenSional solutions. 
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Boxplot of Factor Scores for Factor 1 


Figure 4.3 
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Boxplot of Factor Scores for Factor 2 


Figure 4.4 
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Figure 4.5 Boxplot of Factor Scores for Factor 3 
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Table of Clusters for Five Dimensional Solution 
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V. SUMMARY OF RESULTS 


A. REVIEW OF THE ESSENTIAL ITEMS 


The purpose of this study was to uncover inforajation 
about students perceftions of teacher performance from data 
gathered via a computer interactive survey and SCF fcras. 
The contrel group for the interactive survey was the 
Operations Research section graduating in March of 1985. 
The data consisted of proximity information, how similar or 
dissimilar professors were perceived as being, and rankings 
On bipolar scales. Bipolar scales were chosen fron SOF 
forms and suggestions from previous Operations Research 
students. Separate analyses were conducted on both sets of 
scales. The level of the data was assumed to be interval 
scale in order to utilize the statistical methods involved 
in the analysis. 

Multidimensional scaling was used as the primary means 
of evaluating characteristic differences among professors. 
A monotone relationship among Similarity data was the 
primary Constraint used in determining a final spatial 
Mapping of professors coordinate positions in multidiaen- 
Sional space. The goodness-of-fit criterion used to neasure 
the degree to which the data conformed to the monotonicity 
requirement 1S known as stress. In essence, a value of 
stress between .05 and .1 would indicate a very good fit. 
Unfortunately, the stress value associated with a four 
dimensional interpretation of our data was .242, indicating 
a less than good fit. Other methods helped guide the choice 
of dimensionality. 

Aside from visual inspection of the Spatial mappiags, 


multiple linear regression analysis was used to deteraine 


ane 


the most important characteristics that appeared to set the 
professors apart rrom one another. This was done wey 
regressing the median values of the bipolar scales over the 
coordinate positions of each professor in four space, 
obtained by the MDS program, KYST. The choice of the most 
important characteristics follows from the bipolar scales 
With high multiple correlation coefficients. The multirle 
correlation coefficients associated with all bipolar scales 
were lower than what was hoped for. The correlation coeffi- 
cients were typicaily around .5. For those characteristics 
deemed important, high regression weights determined exactly 
which dimension the associated scale most nearly repre- 
sented. For the bipolar scales used in the interactive 
survey, the scales ‘applied vs theoretical course and 
‘course relied upon prerequisites*t proved to have the 
highest correlation coefficients. These scales most nearly 
explained dimensions 2 and 4. The other two dimensions were 
more difficult to explain since no scale with a high 
multiple correlation coefficient loaded heavily on then. 

For the set of scales obtained from the SOF forns, a 
Similar result occurred with the scales having high multi;le 
correlaticn coefficients loading heavily on only two dimen- 
sions. The scales with the highest correlation coefficients 
here were 1) iastructors objectives made clear, and 2) 


instructor cared about student progress and did his share in 


helping to learn. There seemed to be two indicators coming 
from the SOF forms. One indicator seemed to focus on 
instructor organization and preparation. The “Other aed 


cator involved a student-instructor interaction effect. 
Basically, how did the judges perceive the instructor as 
caring akout the students progress in the course? Thass 
effect seemed to be corroborated in the factor analysis. 

In addition to the bipolar scales, students were asked 


to rate the professors on an overall performance scale. 
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Multiple and stepwise regression efforts fe done using the 
overall performance evaluations obtained from the SOF forms 
and ata later date coincident with taking the comrfuter 
survey. Tne important information gleaned fror the regres- 
Sion analysis suggested that the six bipolar scales used in 
the computer survey did a poor job in explaining students’ 
perceptions of instuctor overall performance. The coeffi- 
client of multiple determination for the regression model was 
a mere .138. Stepwise regression analysis indicated that 
the scale ‘applied vs. theoretical course' accounted for 
most of the variation in the dependent variable, overall 
performance, for its set of scales. Thus, it would seen 
that whether students perceived a course taugkt ty a 
professor as being applied or theoretical had a more signif- 
icant bearing on the overall performance of the professor 
than the other scales in that set. In any case, none o£ the 
scales proved to be statistically significant at the .05 
level. 

The stepwise regression performed on the SCF data 
yielded a coefficient of multiple determination, F2 , of 
Oe This rather high value of K2 seemed to sucgest that 
the scales used in the SOr forms more nearly expiain the 
varlation in overall performance than do the scales used in 
the computer survey. ‘Course organization! accounted for 74 
percent of the explained variation. Even @ith thas high 
value of R2 , =most of the coefficients of the indepencernt 


variables turned out to be statistically insignificant at 


the .05 level. This indicated that high multicollinearity 
existed among the scales. A check for multicollinearity 
proved positive in both sets of scales. Each incevoencent 


variakle was regressed over the other independent variahies 
and high values of Kk? resulted. The multaucollineariss 
problem suggested that a number of the scales, particulariy 


on the SOF forms, be combined into one scale or neasure. 


6 1 


In addition to regression analysis, a factor analysis 
was done on both sets of scales and three separate factors 
were obtained. Factor 1 was composed of the three scales, 
‘grading policy', ‘effort required outsidesclass't, and “pace 
of course’. Apparently, the judges found these scales to 
interact consistently. Factor 2 appeared to describe a 
composite effect between ‘class sizet and ‘course relied 
upon prerequisites’. The factor loading was positive on 
'class size' and neyative on ‘course relied upon preredqui- 
sites'. One might infer that the larger the class size, the 
less that course was perceived as requiring prerequisite 
courses in Operations kesearch. This appeared to be true 
Since most of the larger classes occurred in the beginning 
of the cureivculun Factor 3 seemed to describe a student- 
professor interaction effect. The correlations among the 
variakles in this set were aigh contributing to high factor 
loadings on nearly all variables. 

A disjoint cluster analysis was conducted on the coordi- 
Mates generated from the multidimensional scaling algorithn. 
Each professor was grouped into one and only one cluster. 
Exit interviews with students suggested that five clusters 


appeared to be an appropriate number of groups. 


Be CONCLUSIONS AND RECCOMENDATIONS 


The multidimensional scaling technique seemed to provide 
a weak explanation of instructor groupings. The reasons foc 
this may be several. PLES a linear causal reiationship 
May not be appropriate in describing students' percepticns 
of instructor performance. Certainly the six bipolar scales 
used in the interactive survey were not powerful explanatory 
variables. However, there seem to be one or two strong 
indicators among the scales used in tke SOF fOmUS. 


Specifically, a student-instructorenterac tion err cceumona am 


SZ 


instructor organizational and preparation effect appear 
dominant. What needs to be done in the future is to oktain 
Further scales or characteristics from Operations esearch 
students possibly during exit interviews. Also, “one wight 
suggest that another look be given to changing the current 
SOF form as it stands by combining some of the highly corre- 
lated variables and adding new characteristics that later 
prove meaningful. However, it should be noted that what 
might be considered an important characteristic ia 
describing instructor performance for one student group aay 


prove to be less important for another group. 
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APPENDIX B 
POUR DIMENSIONAL SPATIAL MAPPING OF PROFESSORS POSITIONS 


DIMENS!ICN 2 





DIMENSION 1 


65 


DIMENSION 2 





DIMENSION 1 








b NOISNAINIG 


DIMENSION 1 


67 


DIMENSION 3 





DIMENSION 2 


68 


DIMENSION 4 





DIMENSION 2 


69 


DIMENSION 4 


ad 
° 
t 


DIMENSION 3 


70 





PIVE DIMESSIONAL SPATIAL MAPPING OF PROFESSORS POSITICNS 


DIMENSION 2 





DIMENSION 1 


71 


DIMENSION 3 





= 
an) 


DIMENSION 1 


be 





b NCISNINIG 


DIMENSION 1 


73 


DIMENSION 5 





DIMENSION 1 


74 





¢ NOISNINIG 


DIMENSION 2 


aha 





b NOISNIWIC 


DIMENSION 2 


76 


DIMENSION 5 





DIMENSION 2 


V7 








t NOISNIWIO 


DIMENSION 3 


78 





G NOISN3WIO 


DIMENSION 3 


ie 





G NO'ISNAWG 


DIMENSION 4 


80 


APPENDIZ D 


COMPUTER SOURCE CODE POR INTERACTIVE COMPUTER SURVEY 
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